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Abstract: A statewide evaluation in Idaho addressed whether teachers changed in 
teaching philosophy and technology skills following summer in-service workshops and 
whether these changes sustained themselves in the face of school year and classroom 
realities. The timing of data collection is an important factor to consider when 
interpreting program outcomes. Teachers' beliefs about teaching and their sense of 
competency with technologies varied at three different times: prior to summer 
workshops, after the summer workshops, and after 10 months of subsequent teaching. It 
is critical to distinguish changes that are temporary and due to enthusiasm immediately 
after a workshop from changes that sustain themselves or emerge over time as the initial 
impact of the workshop fades and the reality of the classroom comes to the fore. Guided 
by Katzer’s (1981) “error model" of research, we address time of data collection and 
other sources of error to improve our knowledge of program impacts. 


Project Summary 

Teaching with Technology (TWT) is a statewide professional development program in Idaho funded and 
developed by the J.A. and Kathryn Albertson Foundation (www.jkaf.org). TWT provides intensive summer 
workshops led by teachers who provide follow-up support as Technology Fellows during the school year. Each 
workshop lasts five days and covers basic technology skills, advanced technology applications and several 
constructivist-oriented reform models for teaching. TWT emphasizes the reform of teaching practices, not 
merely the development of technology skills. 

In an earlier study, we concluded that technology skills are positively related to achievement gains, but we 
could not address causality or disentangle the effect of technology use at school or at home (Ravitz, 
Mergendoller & Rush, 2002; BIE, 2002a). To evaluate TWT as a professional development program we used 
multiple indicators of program impact including reported helpfulness of training workshop and changes in 
teachers’ attitudes toward technology; requests for technology training; objectives for student technology use; 
beliefs about teaching and learning; and changes in teachers’ technology skills (BIE, 2002b). This paper 
focuses on the last two indicators - changes in technology skills and beliefs about teaching. In our paper we 
examine 1) how teachers’ responses varied at three different times; 2) how much their responses changed; and 
3) who changed the most. 


Study Methods 

Prior to holding the professional development workshops TWT sent self-report surveys to all registered 
participants via mail; if teachers did not return the surveys they were asked to complete the instrument when 
they entered the workshop. A second wave of surveys was given to teachers as they left the workshop to rate 
their initial responses and changes after five days of intense professional development. A third wave of surveys 
was administered via an online survey submission form 10 months later, i.e., after a year of teaching. There 
were differences in the survey administration and scope, but key variables concerning teaching beliefs and 
technology skills were measured the same way during each administration. The survey included measures of 
teaching beliefs that broadly indicate a constructivist-oriented approach to teaching. These measures were 
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correlated with classroom practices and had strong reliability in a prior national study (Ravitz, Becker & Wong, 
2000, Becker, Ravitz, & Wong, 1999; Ravitz, 1999). For our sample, the measures had similar levels of 
reliability and relationships to practice. 

In order to limit response bias and to maximize the power of our longitudinal data we used paired-t tests to 
determine changes over time. Our treatments of response bias and selection bias improve the accuracy of the 
study, while reducing its generalizability. We do not know how non-participating teachers would have fared 
had they chosen to participate in TWT, or how non-respondents might have changed differently from 
respondents. We rely on replications to improve confidence in our findings. We used effect size differences to 
rate the amount of change in scores among participants between waves (Thompson, 1999). Although the 
number of respondents in Wave 3 was relatively low, the changes they reported were substantial and provide 
the basis for this report. 


Findings 

Participating teachers reported their technology skills increased and their thinking moved in a 
constructivist direction (See BIE, 2002b for additional results). However, the changes that were reported 
immediately after the workshop (Wave 1 and 2 changes) seem to reflect an "enthusiasm effect" so that the 
changes reported at the time the training ended were artificially high. This does not mean there was no real 
change during the five day workshop, just that one must be suspect of these reported changes until they have 
been tested in the classroom. 

What we infer is that a number of constructivist-related reforms are more difficult to implement in the 
classroom than they seem at first. Enthused teachers may report a belief that is consistent with constructivist 
learning theories presented in a workshop, but find this belief less tenable or more difficult to put into practice 
in their actual classrooms. Therefore, the timing of data collection is an important source of error to address in 
future professional development research. In particular, after 10 months in the classroom teachers wanted 
quieter classrooms than they originally had indicated after the training (BIE, 2002b). 

There was also drop in reported technology skills between Wave 2 and Wave 3 in several skills, while 
some other skills continued to increase over time (e.g., attach files to email). By Wave 3 teachers were less 
confident in their skills to create a web page and in their multimedia skills than they were at the end of the 
workshop. Despite the fact that these ended up being the same skills with the greatest gains repotted overall 
(third column), this drop from Wave 2 to Wave 3 seems to represent a “correction" that reflects the complexity 
of these technologies as one delves deeper into their application. 

Discussion 

It is important for professional development programs to show that they have played a part in changing 
teachers’ outlook, skills or practice. More important than teachers' initial willingness to change, however, is 
their ability to sustain that change through a year of teaching in the classroom. Pre-post research designs that do 
not leave time for "reality" to set in may obtain artificially favorable results due to what we are calling an 
“enthusiasm effect.” In contrast, changes in self-report measures that occur after training may show a decrease 
in self-reported changes that represent a real correction based on experience during the school year. Even if the 
magnitude of a change decreases over time, the result may represent a real accomplishment, if the initial 
changes were substantial. 

Because we have allowed some time to pass we are more confident that the overall changes reported 
from Wave 1 to Wave 3 are real. Despite their gains in technology skills, teachers also encountered unforeseen 
difficulties that caused them to reassess and, in effect, downgrade the extent of their skills. This is an issue also 
noted by Rockwell & Kohn (1989). Continued challenges in the area of Web and multimedia tools were 
reflected in requests for further training in these areas. In contrast, regarding use of email, there appears to have 
been continued gains in skills reported (BIE, 2002b). The true difficulty associated with use of different 
applications is something that professional development evaluators have to consider when interpreting self- 
reported technology skills after a workshop. 
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We see the importance of disaggregating teachers’ scores so that self-reported data is presented 
according to the key characteristics of the teachers and schools that may influence these scores. If this is not 
done, bottom line -only analyses can be misleading. We conclude that the impact of the program was greatest 
for teachers in the smaller and lower grade schools, i.e., not in the larger high schools and middle schools. 

It seems to be much easier to change teachers’ technology competencies than to change their beliefs 
about teaching and learning. Our analyses show there were substantial increases in the technology skills 
reported by teachers and only moderate increases in the constructivist beliefs. Only teachers in elementary 
schools and in smaller middle schools reported sustained changes in pedagogical beliefs through Wave 3. The 
least change observed - and consequently, the biggest challenge - was in changing the teaching beliefs of 
teachers in high schools, and larger middle schools. 

We suspect these different outcomes may be less an issue of program design, and more an issue of 
organizational capacity. How can there be such substantial gains in technology skills among teachers in larger 
secondary schools and lack of change in teaching beliefs among these same teachers? This seems to confirm 
earlier studies showing that teachers’ technology skills are a relatively small component of technology reform 
and other conditions are required to see a change for students (Ravitz, 1999; 2002). 

Perhaps larger secondary schools had less of an opportunity to apply what they learned in the 
workshop. As noted in the Opportunity One evaluation, larger secondary schools had relatively low per capita 
school use by students (BIE, 2002a), a concern also noted by Cuban (2001). As a result, while they developed 
their own technology skills (Ravitz, 1999), high school teachers were less likely to change their teaching beliefs 
in a way that might have resulted from successful program implementation (use of reform models supported by 
technology as taught in TWT workshops). High school teachers may be more limited by the constraints of their 
environment, and less free to implement constructivist-oriented changes in teaching, regardless of their own 
technology skill development. 

We are still trying to identify the most important issues to address in professional development 
evaluations. A few researchers have gone further than we have in attempting to measure the relative impact of 
different professional development program components (e.g., Porter, et al., 2000). This is the direction we 
hope to take, once we are more confident w have identified key sources of error and alternative explanations for 
professional development outcomes and technology-supported reforms. 

We feel it has been useful to focus on for whom and under what conditions professional development 
seems to have an impact. Of course, averages for groups do not reflect on individuals and should not draw 
conclusions from group averages when judging individuals, or generalize from experiences with individuals to 
others of the same group (Popham, 2002). There is often a wide range of variation within groups, even when 
there are differences on average across groups. We are interested in findings from others states, particularly 
with larger urban populations, to see how school size may bias the results of professional development towards 
less constructivist-oriented change and more limited use of technology with students. 
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